On Blockchain
__TOC__ General * Blockchain ** a distributed database that maintains a continuously-growing list of records called blocks. * A Next-Generation Smart Contract and Decentralized Application Platform * How Bitcoin Works Under the Hood (July 14, 2013) * Private blockchain * On Public and Private Blockchains (August 7th, 2015) * What are the best open source projects to study blockchain technology in terms of simplicity? * What is the best programming language to learn if you want to work on the blockchain? * Bitcoin mining the hard way: the algorithms, protocols, and bytes * A 101 Noob Intro to Programming Smart Contracts on Ethereum * [https://www.linkedin.com/pulse/blockchain-database-adi-ben-ari Blockchain as a Database] (July 31, 2015) * State of the ÐApps Smart Contract * Smart Contract (on Wikipedia) RAFT * The Raft Consensus Algorithm Bitcoin * Bitcoin Developer Guide * Mastering Bitcoin (O'reilly, Andreas M. Antonopoulos) * Bitcoin Wiki * Proof of Work ** Proof of work ** The Proof-of-Work Concept (June 24, 2013) Ethereum * Ethereum Wiki * A Next-Generation Smart Contract and Decentralized Application Platform * Ethereum Design Rationale * Ethereum Development Tutorial * Ethereum Homestead Documentation ** the reference documentation accompanying the Homestead release of the Ethereum project. * Ethereum Builder's Guide * Ethereum Frontier Guide * Awesome Ethereum Virtual Machine * Ethereum Improvement Proposals (EIPs) ** standards for the Ethereum platform, including core protocol specifications, client APIs, and contract standards * ERC ** Application-level standards and conventions * Enterprise Ethereum Architecture Stack Launched (May 2, 2018) * Concepts ** Ether *** 1 wei = 10-18 ether *** 1 Gwei (shannon) = 109 wei = 10-9 ether ** Account Types, Gas, and Transactions *** Externally Owned Account, Contract Account, Transaction, Message, Gas ** What is Gas? ** What is coinbase? (Apr 16 '16) * Specs ** Ethereum Natural Specification Format ** Ethereum Chain Spec Format *** Subformat: genesis ** Enode URL format * APIs ** JSON RPC API (Generic) *** A transaction object ** JavaScript API ** Management APIs * Clients ** GETH & ETH Command line tools for the Ethereum Network ** How to call “geth attach” on --testnet better than writing full path? (Jan 9 '17) *** geth attach http://127.0.0.1:8545 * Test Network, Private Network ** Running geth *** Full node on the main Ethereum network, Full node on the Ethereum test network, Operating a private network ** Private network ** The Geth’s saga: setting up Ethereum private network on windows (Sep 25, 2017) ** Setting up private network or local cluster ** Running Ethereum testnet using Geth ** Ethereum testnets ** ethereum-private-network *** Instructions on how to setup an ethereum private network ** ADDING NODES TO YOUR PRIVATE BLOCKCHAIN ** [https://ethereum.stackexchange.com/a/2377 Genesis block Explanation] * Keys and Address ** Accounts, Addresses, Public And Private Keys, And Tokens ** Understanding the concept of Private Key, Public Key and Address in Ethereum Blockchain (2017/11/17) ** Location of public and private keys of Ethereum account (Apr 25 '17) *** The private key is never (or should never be) saved unencrypted on disk. It is generated from the info in the keystore file when the user enters the password. ** how to produce enode from node key? (Oct 21 '17) *** bootnode -nodekeyhex 59233b25bfa4c214a8713e07a395a5d11478de10f36c6c80ba5369541f73bc44 -writeaddress ** Create full Ethereum wallet, keypair and address (Feb 15, 2017) ** What does v, r, s in eth_getTransactionByHash mean? (May 3 '17) ** Get public key of any ethereum account (Mar 31 '17) * Programming ** A Gentle Introduction to Ethereum Programming, Part 1 (Nov 15, 2017) ** A Gentle Introduction to Ethereum Programming, Part 2 (Nov 17, 2017) ** A Gentle Introduction to Ethereum Programming, Part 3 (Jan 13, 2018) ** How can I get the data of the latest 10 blocks via web3.js? (Feb 25 '16) *** web3.eth.getBlock(web3.eth.blockNumber - i) * Services ** a Block Explorer and Analytics Platform for Ethereum * Recommendation ** 5 Essential Ethereum Dapp Tools ** 11 Best Ethereum Development Tools to Grow Your Stack (Feb 25, 2017) * misc ** Example genesis.json Go Ethereum * https://github.com/ethereum/go-ethereum * Desc. : Official golang implementation of the Ethereum protocol * License * References ** Go Ethereum Wiki ** geth Command Line Options ** JavaScript Console *** $ geth attach ipc:/some/custom/path, $ geth attach http://191.168.1.1:8545, $ geth attach ws://191.168.1.1:8546 ** JSON RPC API (web3, net, eth, db, ssh) ** JavaScript API ** Management APIs (admin, debug, miner, personal, txpool) * Readings ** Executing custom JavaScript Snippets through Non-Interactive console (geth --exec) (Jun 24 '16) *** geth --exec "personal.unlockAccount('$addr', '$passwd', 0)" --verbosity 4 attach http://127.0.0.1:$rpc_port Ethereumj * https://github.com/ethereum/ethereumj * Desc. : a pure-Java implementation of the Ethereum protocol * License : LGPL v3.0 Parity * https://www.parity.io/ * Desc. : an Ethereum client, written from the ground-up for correctness-verifiability, modularisation, low-footprint and high-performance * License : GPL v3.0 * Written in : Rust * Sources : https://github.com/paritytech/parity * Readings ** Parity official documentation Solidity * https://github.com/ethereum/solidity * Desc. : a contract-oriented, high-level language for implementing smart contracts * License : GPL v3.0 * Sources : https://github.com/ethereum/solidity/ References * Solidity (on Wikipedia) ** an object-oriented programming language for writing smart contracts. * Solidity official documentation ** Data types ** Units and Globally Available Variables ** Block and Transaction Properties ** Time Units ** Address ** Array ** Functions ** Contracts * Standard library (draft) * Change Log * Common Patterns * Ethereum Contract ABI * Solidity Tutorials * Solidity Tutorials * Learn Solidity in Y minutes * How To Learn Solidity: The Ultimate Ethereum Coding Guide Readings Grammar * What is the zero, empty or null value of a struct? (Feb 2 '16) * How to test if a struct state variable is set (Jan 24 '17) * What is the zero value for a string? (Jun 5 '16) Tools * Solidity browser based compiler * Javascript bindings for the solidity compiler ** compile function * What are *C*, *E* and *S* properties in message call return object? (Aug 1 '16) : BigNumber misc * Library Driven Development in Solidity * How do oracle services work under the hood? (Jan 23 '17) Libraries * A place for all the ÐApps to live Examples * Voting * Ethereum Pet Shop Collections * [https://github.com/bkrem/awesome-solidity awesome-solidity] : A curated list of awesome Solidity resources, libraries, tools and more solc-js * JavaScript bindings for the Solidity compiler * Desc. : https://github.com/ethereum/solc-js * License : MIT License web3j * https://web3j.io/ * Desc. : a lightweight, highly modular, reactive, type safe Java and Android library for working with Smart Contracts and integrating with clients (nodes) on the Ethereum network * License : Apache License 2.0 * Written in : Java * Sources ** https://github.com/web3j/web3j ** https://github.com/web3j/quorum ** https://github.com/web3j/examples * References ** web3j official documentation * APIs and Classes extends +------------ Ethereum JsonRpc2_0Web3j ---------------- Web3j ---| implements +------------ Web3jRx extends * Readings ** Deploying and interacting with smart contracts *** Creation of a smart contract *** Transacting with a smart contract *** Querying the state of a smart contract Truffle * http://truffleframework.com/ * Desc. : a development environment, testing framework and asset pipeline for Ethereum, aiming to make life as an Ethereum developer easier * License : MIT * Sources : https://github.com/trufflesuite/truffle * References ** Truffle official documentation ** Command Reference ** [http://truffleframework.com/docs/getting_started/contracts Interacting with your contracts] ** [https://github.com/trufflesuite/truffle-contract#api Truffle Contract API] ** Deployer API ** Assert.sol ** Chai assert API ** Truffle Tutorials * Readings ** The Complete Ethereum Blockchain development Tutorials (Draft) ** Building with Webpack ** “SyntaxError: Unexpected token import” on truffle test (Jul 14 '17) : require Babel ** Ethereum: Deploy on a Private Blockchain - truffle, geth, and private blockchains (07 Mar 2017) ** truffle compile error on windows (29 Jun 2017) : truffle.cmd compile ** USING INFURA (OR A CUSTOM PROVIDER) * Examples ** example-truffle-library : Example project showing how to use Truffle to create javascript libraries ** truffle-webpack-demo : A demo Webpack + React App using truffle-solidity-loader ** truffle-typescript-example : Minimal example integration of Truffle + TypeScript + TestRPC run in docker, for writing smart contracts on Ethereum blockchain. Zeppelin Solidity * https://github.com/OpenZeppelin/zeppelin-solidity * Desc. : a library for writing secure Smart Contracts on Ethereum. * License : MIT * Readings ** Zeppelin-Solidity documentation ** Zeppelin API Embark * https://github.com/iurimatias/embark-framework * Desc. : Framework for serverless Decentralized Applications using Ethereum, IPFS and other platforms * License : MIT Mist Browser * https://github.com/ethereum/mist * Desc. : the tool of choice to browse and use Ðapps Ganache CLI * https://github.com/trufflesuite/ganache-cli * Desc. : simulate full client behavior and make developing Ethereum applications faster, easier, and safer. * License : MIT Remix IDE * https://github.com/ethereum/remix-ide * Desc. : a browser-based compiler and IDE that enables users to build Ethereum contracts with Solidity language and to debug transactions * License : MIT License Oraclize * http://www.oraclize.it/ * Desc. : the leading oracle service for smart contracts and blockchain applications, serving thousands of requests for day every day on Ethereum, Bitcoin and Rootstock. * License : * Readings ** Oraclzie official documentation Quorum * https://www.jpmorgan.com/global/Quorum * Desc. : an Ethereum-based distributed ledger protocol with transaction/contract privacy and new consensus mechanisms * License : LGPL v3.0 + GPL v3.0 * Written in : * Sources : ** https://github.com/jpmorganchase/quorum ** https://github.com/jpmorganchase/constellation * References ** Quorum wiki *** Running Quorum ** Provisioning *** Getting Set Up *** Running Quorum *** Installing and running Constellation *** Constellation Configuration Sample *** Quorum Examples * Readings ** BUILDING DAPPS FOR QUORUM: PRIVATE ENTERPRISE BLOCKCHAINS ** Raft-based consensus for Ethereum/Quorum ** Gas price issue in Quorum Ripple * https://ripple.com/ * Desc. : a network of computers which use the Ripple consensus algorithm to atomically settle and record transactions on a secure distributed database * License : ISC license * Written in : C++ * Sources : https://github.com/ripple * RPM packages : ** https://mirrors.ripple.com/rpm/ ** https://mirrors.ripple.com/rpm/el7/stable/x86_64/ * WebSocket API Tool * Data API v2 Tool References * Ripple Developer Center * Ripple Wiki * Ripple Concepts * rippled API Reference * RippleAPI Reference : the official client library to the XRP Ledger * Ripple Data API : provides access to information about changes in the XRP Ledger, including transaction history and processed analytical data * rippled-example.cfg rippled API Consensus * [https://github.com/ripple/rippled/blob/develop/docs/consensus.md Consensus and Validation] Round Start --------------- Close -------------------- Consensus -------------------- Round End Open phase Establish phase Accept phase * The XRP Ledger Consensus Process * Consensus ** Concensus *** Through the consensus process, validating nodes agree on a specific subset of the candidate transactions to be considered for the next ledger *** Consensus is an iterative process in which nodes relay proposals, or sets of candidate transactions. *** Nodes communicate and update proposals until a supermajority 5 of peers agree on the same set of candidate transactions. ** Validation *** The validating nodes calculate a new version of the ledger and relay their results to the network, each sending a signed hash of the ledger it calculated based on the candidate transactions proposed during consensus. *** In cases where a node is in the minority, having computed a ledger that differs from its peers, the node disregards the ledger it computed 9. It recomputes the correct ledger, or retrieves the correct ledger as needed. * rippled Server States +--> full | disconnected ---> connected ---> syncing ---> tracking --| | +--> validating <-----> proposing Transaction * Reliable Transaction Submission ** [https://developers.ripple.com/reliable-transaction-submission.html#best-practices Best Practices] ** Your rippled server should automatically acquire the missing ledger versions when it has spare resources (CPU/RAM/disk IO) to do so, unless the ledgers are older than its configured amount of history to store. Depending on the size of the gap and the resource usage of your server, acquiring missing ledgers should take a few minutes. You can also manually request your server to acquire historical ledger versions using the ledger_request method. ** Use the LastLedgerSequence parameter to prevent undesirable cases where a transaction is not confirmed promptly but could be included in a future ledger. You should specify the LastLedgerSequence parameter on every transaction. Automated processes should use a value of 4 greater than the last validated ledger index to make sure that a transaction is validated or rejected in a predictable and prompt way. * Transaction submission and verification programming guideline * Transaction Common Fields * Transaction Results Ledger * Ledgers ** Ledger = Header + Transaction Tree + State Tree (Ledger Objects) ** At any given time, a rippled instance has an in-progress "current" open ledger, plus some number of closed ledgers that have not yet been approved by consensus, and any number of historical ledgers that have been validated by consensus. ** Only the validated ledgers are certain to be correct and immutable. * Ledger Data Formats ** boolean ledger.closed ** boolean ledger.accepted * Ledger Header ** Two ledgers with the same hash are always the same. ** For validated ledgers, hash values and sequence numbers are equally valid and correlate 1:1. ** Two different rippled servers may have different contents for a current ledger with the same ledger index, due to latency in propagating transactions throughout the network. ** There may be multiple closed ledger versions competing to be validated by consensus. These ledger versions have the same sequence number but different contents (and different hashes). Only one of these closed ledgers can become validated. ** A current ledger's contents change over time, which would cause its hash to change, even though its ledger index number stays the same. The hash of a ledger is not calculated until the ledger is closed. Amendments * Amendments * Known Amendments Readings * Install rippled Stellar * https://www.stellar.org/ * Desc. : Stellar is a platform that connects banks, payments systems, and people. Integrate to move money quickly, reliably, and at almost no cost. * License : * Sources : https://github.com/stellar * Stellar on Wikipedia Hyperledger * Hyperledger Projects ** Hyperledger Cello ** Hyperledger Fabric ** Hyperledger Iroha ** Hyperledger Sawtooth Lake Hyperledger Fabric * https://github.com/hyperledger/fabric * Desc. : an implementation of blockchain technology, leveraging familiar and proven technologies. * License : Apache License Version 2.0 * Written in : Go * Docker repository : https://hub.docker.com/u/hyperledger/ * Docker images : https://github.com/hyperledger/fabric/tree/master/images * Projects ** hyperledger/fabric ** hyperledger/fabric-docs ** hyperledger/fabric-samples ** hyperledger/fabric-chaincode-java ** hyperledger/fabric-chaincode-node ** hyperledger/fabric-chaintool ** hyperledger/fabric-sdk-rest ** hyperledger/fabric-sdk-node ** hyperledger/fabric-sdk-java ** hyperledger/fabric-sdk-py ** hyperledger/fabric-sdk-go Readings * References ** Official documentation ** Transaction Flow ** Read-Write set semantics ** Bringing up a Kafka-based Ordering Service ** CLI ** Specification ** REST API * Concepts ** Channels *** Channel = Organizations + Ordering Service + Anchor Peers + Chaincodes + Ledger *** At least one anchor peer per organization * Setup ** Getting Setup ** Setting Up a Network ** Multichannel Setup ** Install and Instantiate ** v1.0.0-alpha2 *** sample configtx.yaml *** sample core.yaml *** sample orderer.yaml *** Getting Started ** v1.0.0-beta *** sample crypto-config.yaml * Ordering ** A Kafka-based Ordering Service for Fabric * Ledger ** Hyperledger Fabric V1.0: Block Structure (Part 1) * Chaincode ** What is chaincode? ** Chaincode API ** Java chaincode ** [http://hyperledger-fabric.readthedocs.io/en/latest/Setup/Chaincode-setup.html Writing, Building, and Running Chaincode in a Development Environment] ** Chaincode examples ** Where is Hyperledger fabric chaincode store location? (Jan 26) * SDK ** Hyperledger Fabric SDK Design Specification v1.0 * misc ** Hyperledger Fabric Approaches v1.0 with Better Scalability and Security ** Logging Control ** Hyperledger Fabric V1.0: Block Structure (Part 1) Examples * CLI based End-to-End Flow ** 1 Orderer, 3 Peer, 1 CLI ** configtx.yaml ** docker-compose.yaml ** docker-compose-no-tls.yaml ** script.sh : called by yaml file * Fabric official samples ** Build Your First Network (BYFN) * github.com/ksachdeva/hyperledger-fabric-example ** ksachdeva.github.io Demos * ITPeople Blockchain - Art Auction Demo * IBM Blockchain - Marbles Demo Sources * fabric/core/ledger/util/couchdb/couchdb.go ** 1.0.0-alpha *** handleRequest(method, connectURL string, data io.Reader, rev string, multipartBoundary string) (*http.Response, *DBReturn, error) ** 1.0.0-beta *** handleRequest(method, connectURL string, data [byte, rev string, multipartBoundary string, maxRetries int, keepConnectionOpen bool) (*http.Response, *DBReturn, error)] * fabric/core/chaincode/shim/interfaces.go ** 1.0.0-alpha *** PutState(key string, value [byte) error] *** GetState(key string) ([byte, error)] * fabric/core/ledger/ledgerconfig/ledger_config.go ** 1.0.1 Issues * FAB-6092 : Gossip error: PKIID wasn't found (10/Sep/17) * FAB-6421 : Performance and scale improvements for 1.1 (03/Oct/17) Docker Images fabric-peer Config which is printed by "docker inspect --format=' ' ..." is "Config": { "Hostname": "e7eddde82bec", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "PEER_CFG_PATH=/etc/hyperledger/fabric", "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/fabric/msp/sampleconfig" ], "Cmd": [ "peer", "node", "start" ], "ArgsEscaped": true, "Image": "sha256:793719e9dd193f580f32c5984ac47a8c0f986819e4795c039703b26bb6ad15ce", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": [], "Labels": { "org.hyperledger.fabric.base.version": "0.3.0", "org.hyperledger.fabric.version": "1.0.0-alpha" } }, Fabric CA * Fabric CA User’s Guide * Fabric CA Server REST API Fabric SDK for Node.js * Sources : ** https://github.com/hyperledger/fabric-sdk-node ** https://github.com/hyperledger/fabric-sdk-node/tree/master/fabric-client ** https://github.com/hyperledger/fabric-sdk-node/tree/master/fabric-ca-client * Features ** create channels ** ask peer nodes to join the channel ** install chaincodes in peers ** instantiate chaincodes in a channel ** invoke transactions by calling the chaincode ** query the ledger for transactions or blocks * Readings ** Official documentation ** [https://fabric-sdk-node.github.io/ Application developer documentations] * Examples ** Balance transfer example *** https://github.com/hyperledger/fabric-sdk-node/tree/master/examples/balance-transfer Performance Tuning * JBOD or RAID * LVM (Logical Volume Manager) * Filesystem Journaling ** Disabling Journaling in Ubuntu 14.04? * ulimit, open files ** Set ulimit parameters on ubuntu (2009/06/04) ** Raising the Maximum Number of File Descriptors (Open Files) on Ubuntu 14.04 Trusty (2015/05/18) * noatime ** Reducing Disk IO By Mounting Partitions With noatime ** Is it worth to tune Ext4 with noatime? (Aug 12 '10) ** RHEL 6 Filesystem Mount Options * vm.swappiness ** Swappiness ** RHEL 6 Tuning Virtual Memory *** A low swappiness value is recommended for database workloads. ** Swap Space and Kernel Swappiness ** How do I configure swappiness? (Ask Ubuntu) * Disk I/O Scheduler ** Noop for SSD ** Linux I/O Scheduler Comparison On The Linux 3.4 Desktop(11 May 2012) ** Selecting a Linux I/O Scheduler ** Linux Change The I/O Scheduler For A Hard Disk ** SSD benchmark of I/O schedulers (April 28th, 2010) * Docker OverlayFS storage driver ** Use the OverlayFS storage driver * CouchDB ** Linux tuning for better CouchDB performance * misc ** Tuning Linux for Database performance Companions PTE * https://github.com/dongmingh/v1performance * Desc. : Performance Traffic Engine R3 Corda * https://www.corda.net/ * Desc. : The open source blockchain for business * License : Apache License, Version 2.0 * Written in : Kotlin * Sources : https://github.com/corda/corda Concepts * Key Concepts State Object ----+----> Contract Code +----> Legal Prose Features * All communication between nodes is direct, with TLS-encrypted messages sent over AMQP/1.0 ** AMQP(Advanced Message Queuing Protocol) References * Corda 3.2 Documentation * Start Coding Corda * Setup and run cordapp-example project * The example CorDapp (https://github.com/corda/cordapp-example) * The Corda Platform: An Introduction (May, 2018) * Technical White Paper (Version 0.5, November 29, 2016) * End-State Principles * Intel SGX (Software Guard Extensions) Readings * Building Blocks of the State Machine Approach to Consensus (Jun 20, 2016) * Closed Seal Sets and Truth Lists for Better Privacy and Censorship Resistance (Jun 22, 2016) Eris * https://monax.io/ * Desc. : an open platform for developers and devops to build, ship, and run blockchain-based applications for business ecosystems. * License : https://monax.io/library/#licensing * Sources : https://github.com/eris-ltd * References ** Eris official documentation ** Eris Getting Started *** Building Your First Blockchain App with Eris (August 24, 2016) ** Eris Command Line Interface ** [https://monax.io/docs/documentation/db/latest/specifications/api/ Eris DB Web APIs] ** Jobs Specification ** Variables Specification * Readings ** Using Docker Machine With Eris ** On Eris and Tendermint: Application and Consensus (2016/03/02) ** Cannot deploy smart contract with Eris / Monax *** Could not perform pkg action service: Could not perform pkg action: Docker: {"mssage":"Cannot link to a non running container: /simplechain ** The Blockchain Explained to Web Developers, Part 1: The Theory (2016/04/28) ** The Blockchain Explained to Web Developers, Part 2: In Practice (2016/05/20) ** The Blockchain Explained to Web Developers, Part 3: The Truth (2016/06/14) ** Eris and Ethereum - Decentralized computing on a blockchain (May 20, 2015) Tendermint * https://tendermint.com/ * Desc. : Byzantine fault-tolerant replicated state machines in any programming language * License : ? IPFS * https://ipfs.io/ * Desc. : A peer-to-peer hypermedia protocol to make the web faster, safer, and more open